Configure alert notifications in Splunk App for Infrastructure

Configure an entity or group alert to send a notification when an entity or group meets or exceeds a certain threshold. You can configure these types of alert notifications:

• Email
• VictorOps for Splunk
• Slack webhook
• Custom webhook

You can include multiple alert notification methods for each alert, and mix and match alert notifications for alert thresholds. For example, you can create two alert notifications with different notification methods that share the same alert threshold.

Configure email notifications

SAI uses Splunk Enterprise email notification settings to send email notifications when alerts meet or exceed certain thresholds. For more information about configuring email notification settings, see Email alert action in the Splunk Enterprise Alerting Manual.

Configure VictorOps for Splunk notifications

VictorOps For Splunk is automated incident management software that aligns log management, monitoring, and chat tools to automate the delivery of alert notifications. When you integrate VictorOps with SAI, you can create and manage alerts in VictorOps to notify a designated person or on-call team with information about a triggered SAI alert.

Prerequisites

• You have the admin or sc_admin role. For more information, see Admin and user roles in Splunk App for Infrastructure.
• You have administrator capabilities in VictorOps.

Steps

Follow these steps to integrate SAI notifications with VictorOps.

1. In VictorOps, get your API Key and Routing Key. If you need help finding the API Key and Routing Key, see the Splunk Integration Guide on the VictorOps website.
2. In Splunk Web, open SAI and go to Settings > Notifications.
3. Under VictorOps settings, enter a unique Name to identify the integration. You can't edit the name after you create it. If you want to edit the name, you have to remove the configuration and create a new one.
4. Enter your Splunk VictorOps API Key and Routing Key.
5. Click Save Credentials. When you save the credentials, SAI sends a test notification to your VictorOps timeline.
6. Verify the authentication of SAI in Splunk VictorOps. Go to your Splunk VictorOps timeline and confirm you received a notification from SAI. The test notification looks like this:

   Splunk SII, Info: Test verification integration.
   

7. If you didn't receive this notification, check your API Key and Routing Key and retry saving your credentials again.

Create and send a VictorOps alert notification

Configure an alert notification to send alerts to VictorOps when metrics for entities hit certain thresholds. These steps show you how to receive a VictorOps alert when a Linux host has a CPU utilization of 95%.

For information about creating and sending alert notifications, see Create and modify alerts in Splunk App for Infrastructure.

Prerequisites

• You have the admin or sc_admin role. For more information, see Admin and user roles in Splunk App for Infrastructure.

Steps

1. From the SAI main menu, select the Investigate tab.
2. Select the host you want create an alert for and select the Analysis tab.
3. From the Data panel, open the Metrics dropdown and click the cpu.system metric. The chart appears in the Analysis workspace.
4. From the chart, click Chart Actions and select Create Alert.
5. For the threshold, select If greater than.
6. Enter 95 for the threshold value.
7. For the notification setting, select Notify If the severity degrades.
8. For the notification method, select via VictorOps.
9. Submit the alert notification.

Remove VictorOps credentials

You can have only one integration with VictorOps in SAI at a time. To remove an integration, delete the saved API Key and Routing Key in SAI.

1. In Splunk Web, open SAI and go to Settings > Notifications.
2. Under VictorOps settings, click Remove Credentials.

Configure Slack webhook notifications

When an SAI alert meets or exceeds a certain alert threshold, you can include a Slack webhook to send alert notifications to. You can set a default Slack webhook in SAI notification settings, and you can specify any Slack webhook when you create or edit an alert notification from the Metrics Workspace.

Prerequisites

For information about Slack incoming webhooks, see Incoming Webhooks on the Slack website.

• You have the admin or sc_admin role. For more information, see Admin and user roles in Splunk App for Infrastructure.
• You have a Slack app in your Slack workspace.
• You have a Slack Incoming Webhook for the Slack app.

Steps

Follow these steps to set a default Slack webhook:

1. In Splunk Web, open SAI and go to Settings > Notifications.
2. Under Slack webhook settings, enter a webhook for the Slack URL.
3. When you're done, click Set default Slack webhook URL to save the Slack webhook.

Configure custom webhook notifications

When an SAI alert meets or exceeds a certain alert threshold, you can include a custom webhook to send alert notifications to. You can set a default custom webhook in SAI notification settings, and you can specify any custom webhook when you create or edit an alert notification from the Metrics Workspace.

Follow these steps to set a default custom webhook:

1. In Splunk Web, open SAI and go to Settings > Notifications.
2. Under Custom webhook settings, enter a webhook for the Webhook URL.
3. When you're done, click Set default custom webhook URL to save the custom webhook.

When you configure an alert to send a notification to a custom webhook, SAI sends a POST request that contains this JSON in the body:

alert_severity: String(info/warning/critical)
alert_title: String(name of alert)
metric_name: String
current_value: Float
state_change: String(improve/degrade)
managed_by_type: String(entity/group)
managed_by_value: String(name of entity/group)
trigger_time: String(epoch time)
aggregation_method: String(avg)
split_by: String
split_by_value: String
filters: Dict(String:Dict)
    metric_exclusive: Dict (String:String)
    metric_inclusive: Dict (String:String)
dimensions: Dict
action_url: String
version: String(1)
type: String(alert)